Sampling Methods in Approximate Query Answering Systems

INTRODUCTION In recent years, advances in data collection and management technologies have led to a proliferation of very large databases. These large data repositories are typically created in the hope that through analysis, such as data mining and decision support, they will yield new insights into the data and the real-world processes that created it. In practice, however, while the collection and storage of massive data sets has become relatively straightforward, effective data analysis has proven more difficult to achieve. One reason that data analysis successes have proven elusive is that most analysis queries, by their nature, require aggregation or summarization of large portions of the data being analyzed. For multi-gigabyte data repositories, this means that processing even a single analysis query involves accessing enormous amounts of data, leading to prohibitively expensive running times. This severely limits the feasibility of many types of analysis applications, especially those that depend on timeliness or interactivity. While keeping query response times short is very important in many data mining and decision support applications, exactness in query results is frequently less important. In many cases, " ballpark estimates " are adequate to provide the desired insights about the data, at least in preliminary phases of analysis. For example, knowing the marginal data distributions for each attribute up to 10% error will often be enough to identify top-selling products in a sales database or to determine the best attribute to use at the root of a decision tree. For example, consider the following SQL query: This query seeks to compute the total number of a particular item sold in a sales database, grouped by state. Instead of a time-consuming process that produces completely accurate answers, in some circumstances it may be suitable to produce ball-park estimates, e.g. counts to the nearest thousands. The acceptability of inexact query answers coupled with the necessity for fast query response times has led researchers to investigate approximate query answering techniques (AQA) that sacrifice accuracy to improve running time, typically through some sort of lossy data compression. The general rubric in which most approximate query processing systems operate is as follows: first, during the " pre-processing phase " , some auxiliary data structures, or data synopses, are built SELECT State, COUNT (*) as ItemCount FROM SalesData WHERE ProductName = 'Lawn Mower' GROUP BY State ORDER BY ItemCount DESC over the database; then, during the " runtime phase " , queries …